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Abstract 


In this paper we are concerned with reproducing kernel Hilbert spaces Hx of functions from an 
input space into a Hilbert space Y, an environment appropriate for multi-task learning. The re- 
producing kernel K associated to Hx has its values as operators on Y. Our primary goal here is to 
derive conditions which ensure that the kernel K is universal. This means that on every compact 
subset of the input space, every continuous function with values in Y can be uniformly approx- 
imated by sections of the kernel. We provide various characterizations of universal kernels and 
highlight them with several concrete examples of some practical importance. Our analysis uses 
basic principles of functional analysis and especially the useful notion of vector measures which 
we describe in sufficient detail to clarify our results. 

Keywords: multi-task learning, multi-task kernels, universal approximation, vector-valued repro- 
ducing kernel Hilbert spaces 


1. Introduction 


The problem of studying representations and methods for learning vector-valued functions has re- 
ceived increasing attention in Machine Learning in the recent years. This problem is motivated 
by several applications in which it is required to estimate a vector-valued function from a set of 
input/output data. For example, one is frequently confronted with situations in which multiple su- 
pervised learning tasks must be learned simultaneously. This problem can be framed as that of 
learning a vector-valued function f = (f1, f2,..., fn), where each of its components is a real-valued 
function and corresponds to a particular task. Often, these tasks are dependent on each other in 
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that they share some common underlying structure. By making use of this structure, each task is 
easier to learn. Empirical studies indicate that one can benefit significantly by learning the tasks 
simultaneously as opposed to learning them one by one in isolation (see, e.g., Evgeniou et al., 2005, 
and references therein). 

In this paper, we build upon the recent work of Micchelli et al. (2006) by addressing the issue of 
universality of multi-task kernels. Multi-task kernels were recently discussed in Machine Learning 
context by Micchelli and Pontil (2005), however there is an extensive literature on multi-task kernels 
as there are important both in theory and practice (see Amodei, 1997; Burbea and Masani, 1984; 
Caponnetto and De Vito, 2006; Carmeli et al., 2006; Devinatz, 1960; Lowitzsh, 2005; Reisert and 
Burkhardt, 2007; Vazquez and Walter, 2003, and references therein for more information) 

A multi-task kernel K is the reproducing kernel of a Hilbert space of functions from an input 
space X which takes values in a Hilbert space Y. For example, in the discussion above, Y = R”. 
Generally, the kernel K is defined on X x X and takes values as an operator from Y to itself.! When 
Y is n-dimensional, the kernel K takes values in the set of n x n matrix. The theory of reproducing 
kernel Hilbert spaces (RKHS) as described in Aronszajn (1950) for scalar-valued functions has 
extensions to any vector-valued Y. Specifically, the RKHS is formed by taking the closure of the 
linear span of kernel sections {K(-,x)y, x € X, y € Y}, relative to the RKHS norm. We emphasize 
here that this fact is fundamentally tied to a norm induced by K and is generally non-constructive. 
Here, we are concerned with conditions on the kernel K which ensure that all continuous functions 
from X to Y can be uniformly approximated on any compact subset of X by the linear span of 
kernel sections. 

As far as we are aware, the first paper which addresses this question in Machine Learning 
literature is Steinwart (2001). Steinwart uses the expression universal kernel and we follow that 
terminology here. The problem of identifying universal kernels was also discussed by Poggio et 
al. (2002). One of us was introduced to this problem in a lecture given at City University of Hong 
Kong by Zhou (2003). Subsequently, some aspects of this problem were treated in Micchelli et al. 
(2003) and Micchelli and Pontil (2004) and then in detail in Micchelli et al. (2006). 

The question of identifying universal kernels has a practical basis. We wish to learn a continuous 
target function f : X — Y from a finite number of samples. The learning algorithm used for this 
purpose should be consistent. That is, as the samples size increases, the discrepancy between the 
target function and the function learned from the data should tend to zero. Kernel-based algorithms 
(Schédlkopf and Smola, 2002; Shawe-Taylor and Cristianini, 2004) generally use the representer 
theorem and learn a function in the linear span of kernel sections. Therefore, here we interpret 
consistency to mean that, for any compact subset Z of the input space X and every continuous 
target function f : X — Y, the discrepancy between the target function and the learned function 
goes to zero uniformly on Z as the sample size goes to infinity. It is important to keep in mind 
that our input space is not assumed to be compact. However, we do assume that it is a Hausdorff 
topological space so that there is an abundance of compact subsets, for example any finite subset of 
the input space is compact. 

Consistency in the sense we described above is important in order to study the statistical perfor- 
mance of learning algorithms based on RKHS. For example, Chen et al. (2004) and Steinwart et al. 
(2006) studied statistical analysis of soft margin SVM algorithms, Caponnetto and De Vito (2006) 
gave a detailed analysis of the regularized least-squares algorithm over vector-valued RKHS and 





1. Sometimes, such a kernel is called operator-valued or matrix-valued kernel if Y is infinite of finite dimensional, 
respectively. However, for simplicity sake we adopt the terminology multi-task kernel throughout the paper. 
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proved universal consistency of this algorithm assuming that the kernel is universal and fulfills the 
additional condition that the operators K(x,x) have finite trace. The results in these papers imply 
universal consistency of kernel-based learning algorithms when the considered kernel is universal. 
One more interesting application of universal kernels is described in Gretton et al. (2006). 


This paper is organized as follows. In Section 2, we review the basic definition and properties 
of multi-task kernels, define the notion of universal kernel and describe some examples. In Section 
3, we introduce the notion of feature map associated to a multi-task kernel and show its relevance 
to the question of universality. The main result in this section is Theorem 4, which establishes that 
the closure of the RKHS in the space of continuous functions is the same as the closure of the space 
generated by the feature map. The importance of this result is that universality of a kernel can 
be established directly by considering its features. In Section 4 we provide an alternate proof of 
Theorem 4 which uses the notion of vector measures and discuss ancillary results useful for several 
concrete examples of some practical importance highlighted in Section 5. 




















name notation information 
input space X a Hausdorff topological space 
Z compact subset of X 
xX,t,z elements of Z 
B(Z) Borel o-algebra of Z 
v signed scalar measure 
u vector measure, see Def. 8 
p,q indices running from 1 to n 
i,j indices running from 1 to m 
output space y Hilbert space, with inner product (-,-) y 
Bı unit ball centered at the origin, in Y 
feature space W Hilbert space with inner product (-,-) y 
L(Y, W) all bounded linear operators from Y into W 
L(Y) all bounded linear operators from Y into itself 
A,B elements of L(V) 
L(Y) C L(Y) | subset of positive linear operators 
multi-task kernel K a function from X x X to L(Y), see Def. 1 
Hg reproducing kernel Hilbert space of K 
feature representation | ® mapping from X to L(Y, W) 
C(Z, Y) space of continuous -valued functions on Z 
1 isometric mapping from C(Z, Y) to C(Z x Bı) 
Ck(Z,%) subset of C(Z, Y) generated by K, see Eq. (2) 
lZ, y) subset of C(Z, Y) generated by ®, see Eq. (9) 





Table 1: Notation. 


2. RKHS of Vector- Valued Functions 


In this section, we review the theory of reproducing kernels for Hilbert spaces of vector-valued 
functions as in Micchelli and Pontil (2005) and introduce the notion of universal kernels. 
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We begin by introducing some notation. We let Y be a Hilbert space with inner product (-,-) y 
(we drop the subscript Y when confusion does not arise). The vector-valued functions will take 
values on Y. We denote by £(%) the space of all bounded linear operators from Y into itself, with 
the operator norm ||A|| := supy,)_; ||Ay|], A E€ L(V) and by £4 (9) the set of all bounded, positive 
semi-definite linear operators, that is, A E€ £4(9) provided that, for any y E€ Y, (y,Ay) > 0. We also 
denote, for any A € L(Y), by A* its adjoint. Finally, for every m € N, we define Nm = {1,...,m}. 
Table 1 summarizes the notation used in paper. 


Definition 1 We say that a function K : X x X — L(Y) is a multi-task kernel on X if K(x,t)* = 
K(t,x) for any x,t € X, and it is positive semi-definite, that is, for any m E€ N, {xj: j © Nn} CX 
and {y;: j €Nm} C Y there holds 


E (i, K(xi,x{)ys) = 0. (1) 
i, jENm 


For any t € X and y € Y, we introduce the mapping K;y : X — Y defined, for every x € X by 
(Kiy) (x) := K(x,t)y. In the spirit of Moore-Aronszjain’s theorem, there is a one-to-one correspon- 
dence between the kernel K with property (1) and an RKHS of functions f : X — Y (Aronszajn, 
1950), see also Micchelli and Pontil (2005) and Carmeli et al. (2006). 

Throughout this paper, we assume that the kernel K is continuous relative to the operator norm 
on L(Y). We now return to the formulation of the definition of universal kernel. For this purpose, 
we recall that C(Z,) is the Banach space of continuous -valued continuous function on a com- 
pact subset Z of X with the maximum norm, defined by || f||..,z := supre z || f(x) ||y. We also define, 
for every multi-task kernel K, the subspace of C(Z,%) 


Ck(Z, Y) := Span{ Ky : x € Zy E€ Y}, (2) 
where the closure is relative to the norm in the space C(Z,%). 


Definition 2 We say that a multi-task kernel K is a universal kernel if, for any compact subset Z of 


X, CK(Z, Y) = C(Z, 9). 


In the special case that Y = R”, the kernel function K takes values as n x n matrices. The corre- 
sponding matrix elements can be identified by the formula 


(K(x,t)) pg = (Krep, Kreg), Vx,t E X, 


where ep,eq are the standard coordinate basis in R”, for p,q € Nn. 

In order to describe some of the examples of multi-task kernels below, it is useful to first present 
the following generalization of Schur product of scalar kernels to matrix-valued kernels. For this 
purpose, for any i € Nm we let y; = (Yii, Y2i, <- -, Yni) € R”, so that Equation (1) is equivalent to 


2 > Ypi(K (xi,%;)) pq; = 0- (3) 


i jENm p,qENn 


From the above observation, we conclude that K is a kernel if and only if ((K (xix) pa) as the matrix 
with row index (p,i) € N, x Nm and column index (q, j) € Nn x Nn is positive semi-definite. This 
fact makes possible, as long as the dimension of Y is finite, reducing the proof of some properties 
of operator-valued kernels to the proof of analogous properties of scalar-valued kernels; this process 
is illustrated by the following Proposition. 
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Proposition 3 Let G and K be n x n multi-task kernels. Then, the element-wise product kernel K o 
G:X xX — R” xR” defined, for any x,t € X and p,q E€ Nn, by (K o G(x,t)) 
(K(x,t)) „(G(x,t)) „ is an n x n multi-task kernel. 

Pq pq 


pa` 


Proof We have to check the positive semi-definiteness of K o G. To see this, for any m € N, {y; € 
R” : i € Nm} and {x; € X : i € Nm} we observe that 


2 (Yi, K o G(x; xj) yj) =} } ying (K (xi,x;)) , (GOA) pg (4) 


i jENm Pi qj 


By inequality (3), it follows that the matrix ((K (x;,x oe ) is positive semi-definite as the matrix 


with (p,i) and (q, j) as row and column indices respectively, and so is ((G(x;,x i) a) Applying 
the Schur Lemma (Aronszajn, 1950) to these matrices implies that Equation (4) is nonnegative, and 
hence proves the assertion. E 


We now present some examples of multi-task kernels. They will be used in Section 5 to illustrate 
the general results in Sections 3 and 4. 
The first example is adapted from Micchelli and Pontil (2005). 


Example 1 /f for every j € Nm the function Gj : X x X — R is a scalar kernel and B; € L4(%), 
then the function 
K(x,t)= }, Gj(x,t)Bj, Vx,t EX (5) 
JENm 


is a multi-task kernel. 


The operators B; model smoothness across the components of the vector-valued function. For ex- 
ample, in the context of multi-task learning (see, e.g., Evgeniou et al., 2005, and references therein), 
we set Y = R”, hence B; are n x n matrices. These matrices model the relationships across the 
tasks. Evgeniou et al. (2005) considered kernels of the form (5) with m = 2, Bı a multiple of the 
identity matrix and B a low rank matrix. A specific case for X = Rf is 


(K(x,t)) pg =x: t+ (1 =) 8 pq(x-t)*, p,q €Nn, 


where x- t is the standard inner product in R? and A € [0,1] . This kernel has an interesting in- 
terpretation. Using only the first term on the right hand side of the above equation (A = 1) corre- 
sponds to learning all tasks as the same task, that is, all components of the vector-valued function 
f= (fi,---, fn) are the same function, which will be a linear function since the kernel G, is lin- 
ear. Whereas, using only the second term (A = 0) corresponds to learning independent tasks, that 
is, the components of the function f will be generally different functions. These functions will be 
quadratic since G2 is a quadratic polynomial kernel. Thus, the above kernel combines two heteroge- 
neous kernels to form a more flexible one. By choosing the parameter À appropriately, the learning 
model can be tailored to the data at hand. 

We note that if K is a diagonal matrix-valued kernel, then each component of a vector-valued 
function in the associated RKHS of K can be represented, independently of the other components, 
as a function in the RKHS of a scalar kernel. However, in general, a multi-task kernel will not be 
diagonal and, more importantly, will not be reduced to a diagonal one by linearly transforming the 
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output space. For example, the kernel in Equation (5) cannot be reduced to a diagonal kernel, unless 
all the matrices B;, j € Nm can all be simultaneously transformed into a diagonal matrix. Therefore, 
in general, the component functions share some underlying structure which is reflected by the choice 
of the kernel and cannot be treated as independent objects. This fact is further illustrated by the next 
example. 


Example 2 If Xo is a compact Hausdorff space, for p € Nn, T, is a map from X from Xo (not 
necessary linear) and G : Xo x Xo — R is a scalar kernel, then 


K(x,t) := (G(Tpx,T4t)) p Watex 


n 
P.4= 


is a matrix-valued kernel on X. 


A specific instance of the above example is described by Vazquez and Walter (2003) in the context 
of system identification. It corresponds to the choices that Xọ = X = R and T,(x) = x+ Tp, where 
Tp E R. In this case, the kernel K models “delays” between the components of the vector-valued 
function. Indeed, it is easy to verify that, for this choice, for all f € Hg and p € Ny, 


fox) := (F(x), ep) = ha Tp), VEX 


where h is a scalar-valued function in the RKHS of kernel G. 

Other choices of the map T, are possible and provide interesting extensions of scalar kernels. 
For instance, the choice K (x,t) := (e%°) : p,q € Nn), where 6 = (Gpq) is a positive semi-definite 
matrix suggested by Example 2. Specifically, the eigenvalue decomposition of the matrix © is 
given by o = £}; Muju? and, for any x € X andi € N, the map TË is given by TË x = VÀiltipX. 
Therefore, we obtain that K(x,t) = (Ey eT =T : p,q € Nn) and, so, by Proposition 3, we 
conclude that K is a matrix-valued kernel. 

It is interesting to note, in passing, that, although one would expect the function 


n 


K(x,t) := (e-e) r Vx,t EX (6) 


P:4= 


to be a kernel over X = Rf, we will show later in Section 5 that this is not true, unless all entries of 
the matrix © are the same. 

Our next example called Hessian of Gaussian is motivated by the problem of learning gradients 
(Solak et al., 2002; Mukherjee and Zhou, 2006). In many applications, one wants to learn an un- 
known real-valued function f(x), x= (x!,...,x“) € Rf and its gradient function Vf = (0, f,...,0af) 
where, for any j € Na, dpf denotes the p-th partial derivative of f. Here the outputs y;, denotes 
the observation of derivative of p-th derivative at sample x;. Therefore, this problem is an appealing 
example of multi-task learning: learn the target function and its gradient function jointly. 

To see why this problem is related with the Hessian of Gaussian, we adopt the Gaussian process 
(Rasmussen and Williams, 2006) viewpoint of kernel methods. In this perspective, kernels are 
interpreted as covariance functions of Gaussian prior probability distributions over suitable sets of 
functions. More specifically, the (unknown) target function f is usually assumed as the realizations 
of random variables indexed by its input vectors in a zero-mean Gaussian process. The Gaussian 
process can be fully specified by giving the covariance matrix for any finite set of zero-mean random 
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variables { f(x;) : i € Nm}. The covariance between the functions corresponding to the inputs x; and 


x; can be defined by a given Mercer kernel, for example, the Gaussian kernel G(x) = exp(— LIP) 
with © > 0, that is, 


cov( f (xi), f(%j)) = Gxi —x;). 
Consequently, the covariance between 0, f and 0,f is given by 


cov(Apf (xi), Ogf (x;)) = Apdgcov( F (xi), F(Xj)) = —9p94G (xi — xj). 


This suggests to us to use the Hessian of Gaussian to model the correlation of gradient function V f 
as we present in the following example. 


Example 3 We let Y = X =R", and, for any x = (xp: p E Nn) E€ X, G(x) = exp(— Ilr al ) with o > 0. 
Then, the Hessian matrix of G given by 


K(x,t) := (—(0pdgG)(x—t): p,q E Nn) Vx,t EX 
is a matrix-valued kernel. 


To illustrate our final example we let L? (R) be the Hilbert space of square integrable functions 
on R with the norm ||h||?, := fgh? (x)dx. Moreover, we denote by W'(R) the Sobolev space of 
order one, which is defined as the space of real-valued functions h on R whose norm 


1 
[Allws = (llall + Ii'll)? 
is finite. 


Example 4 Let Y = L? (R), X = R and consider the linear space of functions from R to Y which 


have finite norm 
2 JE 
isi? = f (se in + [PS oe 


Then this is an RKHS with multi-task kernel given, for every x,t € X, by 





(K(x,t)y)(r) = e T e*sly(s)\ds, Yy E Y, rER. 


This example may be appropriate to learn the heat distribution in a medium if we think of x as time. 
Another potential application extends the discussion following Example 1. Specifically, we consider 
the case that the input x represents both time and a task (e.g., the profile identifying a customer) 
and the output is the regression function associated to that task (e.g., the preference function of a 
customer, see Evgeniou et al., 2005, for more information). So, this example may be amenable for 
learning the dynamics of the tasks. 

Further examples for the case that Y = L? (R?) will be provided in Section 5. We also postpone 
to that section the proof of the claims in Examples 1-4 as well as the discussion about the universality 
of the kernels therein. 

We end this section with some remarks. It is well known that universality of kernels is a main hy- 
pothesis in the proof of the consistency of kernel-based learning algorithms. Universal consistency 
of learning algorithms and their error analysis also rely on the capacity of the RKHS. In particular, 
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following the exact procedure for the scalar case in Cucker and Smale (2001), one sufficient condi- 
tion for universal consistency of vector-valued (multi-task) learning algorithms is the compactness 
of the unit ball of vector-valued RKHS relative to the space of continuous vector-valued functions. 
Another alternate sufficient condition was proved in Caponnetto and De Vito (2006) for the regu- 
larized least-squares algorithm over vector-valued RKHS. There, it was assumed that, in addition 
to the universality of the kernel, the trace of the operators K(x,.x) is finite, for every x € X. Clearly, 
both conditions are fulfilled by the multi-task kernels presented above if the output space Y is finite 
dimensional, but they become non trivial in the infinite dimensional case. However, it is not clear to 
the authors whether either of these two conditions is necessary for universal consistency. We hope 
to come back to this problem in the future. 


3. Universal Kernels by Features 


In this section, we prove that a multi-task kernel is universal if and only if its feature representation 
is universal. To explain what we have in mind, we require some additional notation. We let W be 
a Hilbert space and £(%, W) be the set of all bounded linear operators from Y to W. A feature 
representation associated with a multi-task kernel K is a continuous function 


P: X> L(9,W) 


such that, for every x,t € X 

K(x,t) =®*(x)®(0), (7) 
where, we recall, for each x € X, ®*(x) is the adjoint of ®(x) and, therefore, it is in L(W,Y). 
Hence, from now on we call W the feature space. In the case that Y = R, the condition that 
P(x) € L(Y, W) can be merely viewed as saying that P(x) in an element of W. Therefore, at least 
in this case we can rewrite Equation (7) as 


K(x,t) = (®(x), O80) w. (8) 
Another example of practical importance corresponds to the choice W = R* and Y = R”, both 
finite dimensional Euclidean spaces. Here we can identify ®(x) relative to standard basis of W and 
Y with the k x n matrix P(x) = (®,p(x):r € Nk, p E€ Nn), where ®,, are scalar-valued continuous 
functions on X. Therefore, according to (7) the matrix representation of the multi-task kernel K is, 
for each x,t € X, 
(K(x,t)) pq = L P,p(x)Pru lt), p,q E Nn. 
reN, 
Returning to the general case, we emphasize that we assume that the kernel K has the representation 
in Equation (7), although if it corresponds to a compact integral operator, such a representation will 
follow from the spectral theorem and Mercer Theorem (see, e.g., Micchelli et al., 2006). 
Associated with a feature representation as described above is the following closed linear sub- 
space of C(Z,%) 





Co(Z,Y) := {B*(.)w: we Wi, (9) 
where the closure is taken relative to the norm of C(Z,%). The continuity of the functions ®*(-)w 
follows from the assumed continuity of K(-,-) by 

[D (x)w- Dewl < E E) — BO)? bells, 


IE) — DAE) -PO llw 
= ||K(x,x) + K(t,t) — K (x,t) — K (t,x) lllw y 
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Our definition of the phrase “the feature representation is universal” means that Ca(Z, Y) = C(Z,) 
for every compact subset Z of the input space X. The theorem below demonstrates, as we men- 
tioned above, that the kernel K is universal if and only if its feature representation is universal. The 
content of Theorem 4 and of the other results of this Section (Lemmas 5, 6 and Proposition 7) are 
graphically represented by the diagram in Table 2 


Theorem 4 Jf K is a continuous multi-task kernel with feature representation ®, then for every 
compact subset Z of X, we have that Cx(Z,Y) = lZ, Y). 


Proof The theorem follows straightforwardly from Lemmas 5, 6 and Proposition 7, which we 
present below. 
E 


As we know, the feature representation of a given kernel is not unique, therefore we conclude by 
Theorem 4 that if some feature representation of a multi-task kernel is universal then every feature 
representation is universal. 

We shall give two different proofs of this general theorem. The first one will use a technique 
highlighted in Micchelli and Pontil (2005) and will be given in this section. The second proof will 
be given in the next section and uses the notion of vector measure. Both approaches adopt the point 
of view of Micchelli et al. (2006), in which Theorem 4 is proved in the special case that Y = R. 

We now begin to explain in detail our first proof. We denote the unit ball in Y by Bı := {y : 
y€ Y, |iyl| < 1} and let Z be a prescribed compact subset of X. Recall that B, is not compact in 
the norm topology on Y unless Y is finite dimensional. But it is compact in the weak topology 
on Y since Y is a Hilbert space (see, e.g., Yosida, 1980). Remember that a basis for the open 
neighborhood of the origin in the weak topology is a set of the form {y : y E Y, |(y,y;)| < 1,i E Nm}, 
where y1,...,Ym are arbitrary vectors in Y. We put on B, the weak topology and conclude, by 
Tychonoff’s theorem (see, e.g., Folland, 1999, p.136), that the set Z x Bı is also compact in the 
product topology. 

The above observation allows us to associate ‘Y-valued functions defined on Z to scalar-valued 
functions defined on Z x Bı. Specifically, we introduce the map 1 : C(Z,%) — C(Z x Bı) which 
maps any function f € C(Z, Y) to the function i(f) € C(Z x Bı) defined by the action 


Uf) (ay) F@) yo, Vy) E (2x Bi). (10) 


Consequently, it follows that the map 1 is isometric, since 


sup || (x) lly = sup sup |(f(2),¥) 9] = sup sup |t(f) (x,y), 
xEZ xEZ|ly||<1 xEZyEBI 
where the first equality follows by Cauchy-Schwarz inequality. Moreover, we will denote by 
(C(Z,%)) the image of C(Z,9) under the mapping 1. In particular, this space is a closed lin- 
ear subspace of C(Z x Bı) and, hence, a Banach space. 
Similarly, to any multi-task kernel K on Z we associate a scalar kernel J on Z x Bı defined, for 
every (x,y), (t,u) E X x By, as 


J((x,y), (t,u)) = (K(x,t)u, y). (11) 


Moreover, we denote by Cy (Z x Bı) the closure in C(Z x Bı) of the set of the sections of the kernel, 
{J((x,y),(-,:)) : (x,y) E€ Zx By}. It is important to realize that whenever K is a valid multi-task 
kernel, then J is a valid scalar kernel. 
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Cy(Zx B) = C(Zx 8B) 
I 
Cly) = C499) 


Table 2: The top equality is Proposition 7, the bottom equality is Theorem 4 and the left and right 
arrows are Lemma 5 and 6, respectively. 


The lemma below relates the set Ck (Z, Y) to the corresponding set Cy (Z x Bı) for the kernel J 
on Z x By. 


Lemma 5 Jf Z is a compact subset of X and K is a continuous multi-task kernel then \(Cx(Z,)) = 
CZ x Bı). 


Proof The assertion follows by Equation (11) and the continuity of the map 1. E 


In order to prove Theorem 4, we also need to provide a similar lemma for the set Co(Z,%). 
Before we state the lemma, we note that knowing the features of the multi-task kernel K leads us to 
the features for the scalar-kernel J associated to K. Specifically, for every (x,y), (t,u) E€ X x Bi, we 
have that 


I(x y), (t,4)) = (Py), Pu) w, (12) 
where the continuous function ¥ : X x Bı — W is defined as 


P(x,y) = P(x)y, xE X,y € By. 


Thus, Equation (12) parallels Equation (8) except that X is replaced by X x By. We also denote by 
Cy(Z x Bi) = {(¥(-),w) yi w E W}, the closed linear subspace of C(Z x Bi). 





Lemma 6 Jf Z is a compact subset of X and K is a continuous multi-task kernel with feature 
representation ® then \(Ca(Z,7)) = Cp(Z x Bı). 


Proof The proof is immediate. Indeed, for each x € X, w E W, y E€ Y, we have that (@*(x)w,y)y = 
(wm, P(x) y) qv = (PO,y),) wy. m 


To complete the proof of Theorem 4, as illustrated in Table 2 it suffices to show that Cy (Z x 
Bi) = C)(Z~x Bı). To this end, we review some facts about signed measures and bounded linear 
functionals on continuous functions. Let Q be any prescribed compact Hausdorff space and C(Q) 
be the space of all real-valued continuous functions with norm ||- || 2. We also use the notation 
¥(Q) to denote the Borel o-algebra on Q. Now, we recall the description of the dual space of 
C(Q). By the Riesz representation theorem, any linear functional L in the dual space of C(Q) is 
uniquely identified as a regular signed Borel measure v on Q (see, e.g., Folland 1999), that is, 


Lle) = f sodva), Yge clo). 
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The variation of v is given, for any E € @(Q), by 


\v|(E) := sup { $ |v(Aj)| : {A}; : j € N} pairwise disjoint and UjenAj = E}. 
jeN 
Moreover, we have that ||L|| = ||v||, where ||v|| = |v|(Q) and ||Z|| is the operator norm of L defined 
by ||L|| = supj.y., .~1 L(g) |- Recall that a Borel measure v is regular if, for any E € B(X), 


V(E) =inf{v(U) : E CU,U open} = sup{v(U) : U CE,U compact}. 


In particular, every finite Borel measure on Q is regular, see Folland (1999, p.217). We denote by 
M (Q) the space of all regular signed measures on Q with total variation norm. We emphasize here 
that the Riesz representation theorem stated above requires the compactness of the underlying space 
Q. 

As mentioned above, Z x Bı is compact relative to the weak topology if Z is compact. This 
enables us to use the Riesz representation theorem on the underlying space Q = Z x Bı to show the 
following proposition. 


Proposition 7 For any compact set Z C X, and any continuous multi-task kernel K with feature 
representation ®, we have that Cp(Z x By) = Cy(Zx Bi). 


Proof For any compact set Z C X, recall that Z x Bı is compact if Bı is endowed with the weak 
topology of Y. Hence, the result follows by applying Theorem 4 in Micchelli et al. (2006) to the 
scalar kernel J on the set Z x Bı with the feature representation given by Equation (12). However, 
for the convenience of the reader we review the steps of the argument used to prove that theorem. 
The basic idea is the observation that two closed subspaces of a Banach space are equal if and only if 
whenever a continuous linear functional vanishes on either one of the subspaces, it must also vanish 
on the other one. This is a consequence of the Hahn-Banach Theorem (see, e.g., Lax, 2002). In the 
case at hand, we know by the Riesz Representation Theorem that all continuous linear functionals 
Lon C(Z x Bı) are given by a regular signed Borel measure v, that is for every F € C(Z x Bı), we 
have that 


LF) = ff Fees)avexy): 


Now, suppose that L vanishes on C;(Z x Bı), then we conclude, by (12), that 


jz L 2 i, p E62) (EH) pda, y)aV 0) 


Also, since K is assumed to be continuous relative to the operator norm and Z is compact we 
have that |x, lly = Eyy = (K(x,x)y,y) < supe z||K(x,x)|| < œ. This together with the 
equation 








Wonslavoay)| < f IMoro)avex)lavOxy) < sup EVI B1) 
Zx By wW Zx Bı x. 


imply that the integrand J... ¥(x,y)dv(x,y) exists. Consequently, it follows that 


2 


(x, y)dv(x,y) (13) 
Ww 














Jo fog, LEED MO) avd (e,y)dV(6,0) = 
ZxB, J Zx By Zx By 
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and, so, we conclude that 


¥(x,y)dv(x,y) = 0. (14) 
BX Bı 
The proof of Equation (13) and the interpretation of the W-valued integral appearing in Equation 
(14) is explained in detail in Micchelli et al. (2006). So, we conclude that L vanishes on Cy(Z x Bı). 
Conversely, if L vanishes on Cp(Z x B) then, for any x € Z,y € By, we have that 


J, „EEV n) = (xe), Yeu)vlu)) =0 


Zx By 


that is, L vanishes on C)(Z x Bı). E 


4. Further Perspectives for Universality 


In this section, we provide an alternate proof of Theorem 4 using the notion of vector measure and 
also highlight the notion of the annihilator of a set, a useful tool for our examples of multi-task 
kernels in Section 5. 

At first glance, the reduction of the question of when a multi-task kernel is universal to the 
scalar case, as explained in Section 3, seems compelling. However, there are several reasons to 
explore alternate approaches to this problem. Firstly, from a practical point of view, if we view 
multi-task learning as a scalar problem we may loose the ability to understand cross task interac- 
tions. Secondly, only one tool to resolve a problem may limit the possibility of success. Finally, 
as we demonstrated in Section 3 universality of multi-task kernels concerns density in the subspace 
C)(Z x Bı), not the full space C(Z x Bı), an issue heretofore not considered. Therefore, we can- 
not directly employ the methods of the scalar case as presented in (Micchelli et al., 2003) to the 
multi-task case. 

As we shall see in this section, the concept of vector measure allows us to directly confront the 
set C(Z, Y) rather than following a circuitous path to C;(Z x B1). However, the basic principle 
which we employ is the same, namely, two closed linear subspaces of a Banach space are equal 
if and only if whenever a bounded linear functional vanishes on one of them, it also vanishes on 
the other one. Thus, to implement this principle we are led to consider the dual space of C(Z,). 
We remark, in passing, that this space also arose in the context of the feature space perspective 
for learning the kernel, see Micchelli and Pontil (2005a). For a description of the dual space of 
C(Z,%), we need the notion of vector measures and in this regard rely upon the information about 
them in Diestel and Uhl, Jr. (1977). 

To introduce our first definition, recall that throughout this paper X denotes a Hausdorff space, 
ZC X any compact subset of X and A(Z) the Borel o-algebra of Z. 


Definition 8 A map u : &(Z) — Y is called a Borel vector measure if u is countably additive, that 
is, u(U?_) Ej) = Yj M(E;) in the norm of Y, for all sequences {Ej : j € N} of pairwise disjoint 
sets in B(Z) 


It is important to note that the definition of vector measure given in Diestel and Uhl, Jr. (1977) 


only requires it to be finitely additive. For our purpose here, we only use countably additive mea- 
sures and thus do not require the more general setting used in Diestel and Uhl, Jr. (1977). 
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For any vector measure u, the variation of u is defined, for any E € 4(Z), by the equation 


|u|(E) := ) = up| >: \|u(A;)|| : {A;: j E€ N} pairwise disjoint and UjenAj = e), 
JEN 


In our terminology we conclude from (Diestel and Uhl, Jr., 1977, p.3) that u is a vector measure if 
and only if the corresponding variation |u| is a scalar measure as explained in Section 3. Whenever 
|u|(Z) < œ, we call u a vector measure of bounded variation on Z. Moreover, we say that a Borel 
vector measure u on Z is regular if its variation measure |u| is regular as defined in Section 3. 
We denote by M(Z,) the Banach space of all vector measures with bounded variation and norm 


lull := |u|(4). 
For any scalar measure v € M (Z x Bı), we define a Y-valued function on 4(Z), by the equa- 
tion 


u(E) = i. IWE) VEE B(Z). (15) 


Let us confirm that u is indeed a vector measure. For this purpose, choose any sequence of pairwise 
disjoint subsets {E} : j € N} C Z(Z), and observe that 


L IMEI < = | f f, dy(xy)| < (Sx Bi), 


which implies that |u|(Z) is finite and, hence, u is a regular vector measure. This observation 
suggests that we define, for any f € C(Z,%), the integral of f relative to u as 


[ew D= f fro) y)dv(x,y). (16) 


Alternatively, by the standard techniques of measure theory, for any vector measure u E M (Z, Y) 
the integral f (f(x), du(x)) is well-defined. 
Z 


One of our goals below is to show that given any vector measure u, there corresponds a scalar 
measure v such that Equation (16) still holds. Before doing so, let us point out a useful property 
about the integral appearing in the left hand side of Equation (16). Specifically, for any y E€ Y, we 
associate to any u E M(Z,9), a scalar measure uy defined, for any E € A(Z), by the equation 
Hy(E) := (y,u(E)). Therefore, we conclude, for any f € C(Z), that 


[ore aues)) f Fda) 


To prepare for our description of the dual space of C(Z, Y), we introduce, for each f € C(Z,%), 


a linear functional L,, defined by, 
Lyf = | Fua); (17) 


Then, we have the following useful lemmas, see the appendix for their proofs. 


Lemma 9 If u € M(Z,%) then Ly € C* (Z, Y) and ||Lull = llall 
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Lemma 10 (Dinculeanu-Singer) For any compact set Z C X, the map u> L, is an isomorphism 


from M(Z, Y) to C*(Z,%). 


Lemma 10 is a vector-valued form of the Riesz representation theorem called Dinculeanu-Singer 
theorem, (see, e.g., Diestel and Uhl, Jr., 1977, p.182). For completeness, we provide a self-contained 
proof in the appendix. 

It is interesting to remark that, for any u E€ M (Z, Y) we have established in the proof of Lemma 
10 that there exists a regular scalar measure v on Z x Bı such that 


Laf = (f (x), y)dv(x,y). 


BX Bı 


Since we established the isometry between C*(Z, Y) and M (Z, Y), it follows that, for every regular 
vector measure there corresponds a scalar measure on Z x Bı for which Equation (15) holds true. 
In order to provide our alternate proof of Theorem 4, we need to attend to one further issue. 


Specifically, we need to define the integral J K(t,x)(du(x)) as an element in Y. For this purpose, 
for any uc M(Z,Y) and t € Z we define a linear functional L; on Y at y € Y as 


Ly:= : (K(x,t)y,du(s)). 


Since its norm has the property ||L;|| < (sup,ez||K(x,4)]|) lul], by the Riesz representation lemma, 
we conclude that there exists a unique element ~y in Y such that 


[Kd = (9). 


It is this vector ~y which we denote by the integral ip K(t,x)(du(x)). 


Similarly, we define the integral / P(x) (du(x)) as an element in W. To do this, we note that 
Z 


IEI = || @* (x) |] and ||@*(x)y||? = (K(x, x)y, y}. Hence, we conclude that there exists a constant 


). 
x such that, for all x € X, ||®(x)|| < ||K(x,x) 2 < K. Consequently, the linear functional L on W 
defined, for any w € W, by 


Lo I (© (@)w,du(x)) 


satisfies the inequality ||L|| < ||u||. Hence, we conclude that there exists a unique element w E€ W 
such that L(w) = ( w,w) for any w E€ W. Now, we denote w by ®(x)(du(x)) which means that 
Z 


( [2 (auex)),w) = f (wda). (18) 


We have now assembled all the necessary properties of vector measures to provide an alternate 
proof of Theorem 4. 
Alternate Proof of Theorem 4. We see from the feature representation (7) that 


[Kes (du(x ) = Lowe x)(dux)) = P0) ( | Ddu), Wez. 
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From this equation, we easily see that if f (x) (du(x)) = 0 then, for every t € Z, i K(t,x)(du(x)) = 
Z Z 


0. On the other hand, applying (18) with the choice w = 1 P(x) (du(x)) we get 
Z 


L (© f e)a) = || f woa 


Ww 


Therefore, if, for any t € Z, J KE» ldu) = 0 then f Slaw) = 0, or equivalently, by 
Z Z 
Equation (18), 
f (D (xw, dux) =0, Wwe W. 
Z 


Consequently, a linear functional vanishes on Cx(Z,”) if and only if it vanishes on Cp(Z,%) and 
thus, we obtained that Ck(Z, Y) = Cal 2, Y). E 

We end this section with a review of our approach to the question of universality of multi-task 
kernels. The principal tool we employ is a notion of functional analysis referred to as the annihilator 
set. Recall the notion of the annihilator of a set V which is defined by 


Vi = fue M(Z, Y) : [0da =0,w € V}, 


Notice that the annihilator of the closed linear span of V is the same as that of V. Consequently, 
by applying the basic principle stated at the beginning of this section , we conclude that the linear 
span of V is dense in C(Z, Y), that is, span( V) = C(Z, Y) if and only if the annihilator V+ = {0}. 
Hence, applying this observation to the set of kernel sections K(Z) := {K(-,x)y: xE Z, ye V} or 
to the set of its corresponding feature sections ®(Z) := {®*(-)w: w E W}, we obtain from Lemma 
10 and Theorem 4, the summary of our main result. 


Theorem 11 Let Z be a compact subset of X, K a continuous multi-task kernel, and ® its feature 
representation. Then, the following statements are equivalent. 


1. Ck(Z, Y) = C(Z,). 
2. @lZ, Y) = C3, V). 


3. K(Z)+= fu EM(Z, Y) : f-K(t,x)(du(x))=0, Vt € z) = {0}. 





4. (2) = {uE M(Z,9) : [zP(x)(du(x)) =0} = {0}. 


5. Universal Kernels 


In this section, we prove the universality of some kernels, based on Theorem 11 developed above. 
Specifically, the examples highlighted in Section 2 will be discussed in detail. 

Kernel’s universality is a main hypothesis in the proof of consistency of learning algorithms. 
Universal consistency of the regularized least-squares algorithm over vector-valued RKHS was 
proved in Caponnetto and De Vito (2006); there, it was assumed that, in addition to universality 
of the kernel, the trace of the operators K(x,x) is finite. In particular, this extra condition on the 
kernel holds, for the Example 1 highlighted in Section 2, when the operators B; are trace class, 
and does not hold for Example 4. It is not clear to the authors whether the finite trace condition is 
necessary for consistency. 
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5.1 Product of Scalar Kernels and Operators 


Our first example is produced by coupling a scalar kernel with an operator in £, (9). Given a scalar 
kernel G on X and an operator B € £4(%), we define the function K : X x X — L(Y) by 


K(x,t) = G(x,t)B, Vx,tEXx. (19) 


For any {xj E€ X: j © Nm} and {yj E Y : j E Nn}, we know that (G(x;,x;))i,jen,, and ((Byi,¥j))i,jen, 
are positive semi-definite. Applying Schur’s lemma implies that the matrix (G(x;,x;)(Byi,¥))i,jeN, 
is positive semi-definite and hence, K is positive semi-definite. Moreover, K*(x,t) = K(x,t) = 
K(t,x) for any x,t € X. Therefore, we conclude by Definition 1 that K is a multi-task kernel. 

Our goal below is to use the feature representation of the scalar kernel G to introduce the corre- 
sponding one for kernel K. To this end, we first let W be a Hilbert space and ọ : X — W a feature 
map of the scalar kernel G, so that 


G(x,t) = (0(x),0(t)) gy, Vx,t EX. 


Then, we introduce the tensor vector space W QY. Algebraically, this vector space is spanned by 
elements of the form w @y with w € W, y € Y. For any w1 ®y1,w2@y2 E WQ) Y and c € R, there 
holds the multi-linear relation 


CWOy=w@cy=c(w@y), (wi +w) 9y =w Qy +w 8y, 


and 
w8 (yi +y2) =w@yi +W8Q y2. 


We can turn the tensor space into an inner product space by defining, for any w1 ® y1, W2 ®@ y2 € 


WY, 
(wi @y1,W2@y2) = (w1,W2) (1,92) 


and extending by linearity. Finally, taking the completion under this inner product, the vector space 
WR Y becomes a Hilbert space. Furthermore, if W and Y have orthonormal bases {w; : i € N} 
and {y; : i € N} respectively, then W QY is exactly the Hilbert space spanned by the orthonormal 
basis {w; ® yj : i, j € N} under the inner product defined above. For instance, if W = R* and 
Y =R”, then WRY = R”. 

The above tensor product suggests that we define the map ® : X > L(Y,W@Y) of kernel K 
by 

B(x)y := 0(x) @ VBy, YxEX, VEY, 


and it follows that b* : X —> L(W@Y,%) is given by 
&*(x)(w@y) := (0(x),w) WV By, VxeX,we W, andyeY. (20) 


From the above observation, it is easy to check, for any x,t € X and y,u € Y, that (K(x,t)y,u) = 
(®(x)y, P(t)u). Therefore, we conclude that ® is a feature map for the multi-task kernel K. 

Finally, we say that an operator B € L4 (Y) is positive definite if (By,y) is positive whenever y 
is nonzero. We are now ready to present the result on universality of kernel K. 


Theorem 12 Let G: X x X — R be a continuous scalar kernel, B € £L4(9) and K be defined by 
Equation (19). Then, K is a multi-task universal kernel if and only if the scalar kernel G is universal 
and the operator B is positive definite. 
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Proof By Theorem 11 and the feature representation (20), we only need to show that ®(Z)+ = {0} 
if and only if G is universal and the operator B is positive definite. 

We begin with the sufficiency. Suppose that there exists a nonzero vector measure u such that, 
for any w&y E W 8 Y, there holds 


[E wey). dle) f Oww Badula) = 0. 2D 


Here, with a little abuse of notation we interpret, for a fixed y € Y, (/By,du(x)) as a scalar measure 
defined, for any E € &(Z), by 


f; By, dula) = (VBy.u(E)). 


Since u € M (Z, Y), (WBy,du(x)) is a regular signed scalar measure. Therefore, we see from (21) 
that (v By, du(x)) € ¢(Z)+ for any y € Y. Remember that G is universal if and only if o(Z)+ = {0}, 
and thus we conclude from (21) that (V/By,du(x)) = 0 for any y € Y. It follows that (vVBy, u(E)) = 0 
for any y € Y and E € Z(Z). Thus, for any fixed set E taking the choice y = /Bu(E) implies 
that (Bu(E),u(E)) = 0. Since E is arbitrary, this means u = 0 and thus finishes the proof for the 
sufficiency. 

To prove the necessity, suppose first that G is not universal and hence, we know that, for some 
compact subset Z of X, there exists a nonzero scalar measure v € M (Z) such that v € ¢(Z)+, that 


is, i ((x),w)dv(x) = 0 for any w E€ W. This suggests to us to choose, for a nonzero yo € Y, the 
Z 


nonzero vector measure u = yov defined by u(E) := yoy (E) for any E € A(Z). Hence, the integral 
in Equation (21) equals 


[ {2° @y), ula) = (VByyo) (66) w)avdv(x) = 0. 


Therefore, we conclude that there exists a nonzero vector measure u € &(Z)+, which implies that 
K is not universal. 

If B is not positive definite, namely, there exists a nonzero element yı € Y such that (By;,y,) = 
0. However, we observe that || By, ||? = (By1,y1) which implies that V By; = 0. This suggests to 
us to choose a nonzero vector measure u = y,V with some nonzero scalar measure v. Therefore, we 
conclude, for any w € W and y € Y, that 


[@' wey). dulo)) = VB) [ Owae) 
= (-VBy1) | Q), »)avdv(x) = 0, 


Z 


which implies that the nonzero vector measure u € &(Z)+. This finishes the proof of the theorem. W 


In the special case Y = R”, the operator B is an n x n positive semi-definite matrix. Then, 
Theorem 12 tells us that the matrix-valued kernel K(x,t) := G(x,t)B is universal if and only if G is 
universal and the matrix B is of full rank. 
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We now proceed further and consider kernels produced by a finite combination of scalar kernels 
and operators. Specifically, we consider, for any j € Nm, that G; : X x X — R be a scalar kernel 
and Bj € L+ (Y). We are interested in the kernel defined, for any x,t € X, by 


K(x,t):= }, Gj(x,t)B;. 
JEN, 


Suppose also, for each scalar kernel G;, that there exists a Hilbert feature space W and a feature 
map 0; : X — W). 

To explain the associated feature map of kernel K, we need to define its feature space. For this 
purpose, let H; be a Hilbert space with inner products (-,-) ; for j € Nm and we introduce the direct 
sum Hilbert space © jen,,H; as follows. The elements in this space are of the form (h1, ..., Am) with 
hj € Hj, and its inner product is defined, for any (11,...,/m),(h{,---,) E Djen Hj, by 


((hi,-.-,hm), (his. -shm = $, hg he 


JENm 


This observation suggests to us to define the feature space of kernel K by the direct sum Hilbert 
space W := @ jenn (W; Q Y), and its the map ®: X — L(Y, W), for any x € X and y € Y, by 


D(x)y := (p1 (x) 8 VBL y,---,Om(x) D VBm y). (22) 


Hence, its adjoint operator ®* : X — L(Pjen,, (WiO Y), Y) is given, for any (w1 @y1,...,Wm Q 
Ym) E Bjen, (Wj @ V), by 


D*(x)((wi @y1,---,Wm@yYm)) = Yo (W, ws) wv, VB; Y3. 


JEN 


Using the above observation, it is easy to see that, for any x,t € X, K(x,t) = B*(x)®(t). Thus K is 
a multi-task kernel and ® is a feature map of K. 
We are now in a position to state the result about the universality of the kernel K. 


Theorem 13 Suppose that Gj : X x X — R is a continuous scalar universal kernel, and B; € 
L(Y) for j © Nm. Then, K(x,t) := Lien, Gj(x,t)B; is universal if and only if X jen,, Bj is pos- 
itive definite. 


Proof Following Theorem 11, we need to prove that ®(Z)+ = {0} for any compact set Z if and 
only if £ jcy,, Bj is positive definite. To see this, observe that u € ®(Z)* implies, for any (w1 & 
yl,- Ds Wm Q Ym) € Djen, (WIN), that 


L E 00w) VE yda) = 0. 


JENm 
Since w; € W; is arbitrary, the above equation is equivalent to 
[61 dan (VE; yj,du(x)) =0, Yw; E€ Wj,yj € Y and j € Nm, (23) 


which implies that (,\/B; y;,du(x)) € o(Z)+ for any j € Nm. Recall that G; is universal if and only 
if (Z)+ = {0}. Therefore, Equation (23) holds true if and only if 


(u(E), /Bj yj) = (\/Bjm(E),y/) =0, VEE BZ), yj EY, j ENm (24) 
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To move on to the next step, we will show that Equation (24) is true if and only if 
(u(E),Bju(E)) =0, VEE AZ), j E€ Nm. (25) 


To see this, we observe, for any j € Nm, that || \/Bju(E)||? = (u(E),B ju(E)). Hence, Equation (25) 
implies Equation (24). Conversely, applying Equation (24) with the choice y; = u(E) directly yields 
Equation (25). 

Moreover, we know, for any y € Y and j € Nm, that (B ;y, y) is nonnegative. Therefore, Equation 
(25) is equivalent to that 


(CÈ, Bi)u(E),u(E)) =0, VE € A(Z). (26) 
JENm 


Therefore, we conclude that u € &(Z)+ if and only if the above equation holds true. 
Obviously, by Equation (26), we see that if } jen, Bj is positive definite then u = 0. This means 
that kernel K is universal. Suppose that )’jcy,, Bj is not positive definite, that is, there exists a 


1 
nonzero yo € Y such that || (£ jen, Bj) 7yoll” = ((Ljen,, By)¥0.Y0) = 0. Hence, choosing a nonzero 
vector measure u := yoV, with v a nonzero scalar measure, implies that Equation (26) holds true 
and, thus kernel K is not universal. This finishes the proof of the theorem. a 


Now we are in a position to analyze Examples 1 and 4 given in the Section 2. Since the function 
K considered in Example 1 is in the form of (22), we conclude that it is a multi-task kernel. 

We now discuss a class of kernels which includes that presented in Example 4. To this end, 
we use the notation Z} = {0}UN and, for any smooth function f : R” — R and index a = 


(O1,--., Om) E Z3, we denote the &-th partial derivative by 0% f(x) := ati) 


AL x7... Xn 
that the Sobolev space W* with integer order k is the space of real valued functions with norm 
defined by 


Then, recall 


I= E f Blax, Q7) 


|o|<k 


where || = X jen, Œj, see Stein (1970). This space can be extended to any fractional index s > 0. 
To see this, we need the Fourier transform defined, for any f € L! (R”), as 


JO = f PMO falar, VEER", 


see Stein (1970). It has a natural extension to L?(IR”) satisfying the Plancherel formula || || PR") = 
IA llz2qem- In particular, we observe, for any 0 = (01, ... , Om) € Z and § = (&1,..-,&m) E€ R”, that 


auf (E) = f(E)(2miE1)™ ... (2niEm)°". Hence, by Plancherel formula, we see, for any f € W* with 
k € N, that its norm || f|| w« is equivalent to 


(fc +4nigiPyFi7e rat) 


This observation suggests to us to introduce fractional Sobolev space W* (see, e.g., Stein, 1970) 
with any order s > 0 with norm defined, for any function f, by 


Illi =f Fam SIP) LAG) Pa, 


1633 


CAPONNETTO, MICCHELLI, PONTIL AND YING 


Finally, we need the Sobolev embedding lemma which states that, for any s > 4, there exists an 
absolute constant c such that, for any f € W* and any x € R”, there holds 


[f(x)| <ellfllws, 


(see, e.g., Folland, 1999; Stein, 1970). 
The next result extends Example 4 to multivariate functions. 


Proposition 14 Let Y = L? (R3), X =R and H be the space of real-valued functions with norm 


e= f [fe a t Ee] 


Then this is an RKHS with universal multi-task kernel given, for every x,t € X by 





(K(x,t)y)(r) = e"l f, eN ly(s)ds, WEN r ERS. (28) 
Proof For any fixed t € Rf, it follows from the Sobolev embedding lemma that 


æD] SellfCOllw- 


Combining this with the definition of Sobolev space W! given by Equation (27), we have that 
2 2 2 
OB <e f Dnd , 
= 42 2 f 2 2J] f]]2 
=. I. (JID +n ar)adx < elif’. 


Since, for any y € Bı and x E R, |(y, f(x))y| < blio IF) lly < Illy, by the above equation 
there exists a constant c’ such that, for any y € B, x€ Rand f€ H, 


(FEl < If 


Hence, by the Riesz representation lemma, H is an RKHS (Micchelli and Pontil, 2005). 
Next, we confirm Equation (28) is the kernel associated to H. To this end, it suffices to show 
that the reproducing property holds, that is, for any f € H, ye Y and xE X 


Fy = (Ky, f), (29) 


where (-,-) denotes the inner product in H. 
By the Plancherel formula, we observe that the left-hand side of Equation (29) equals 


fsa fem nil) F(E ¢) dé Jac= f p30 on 2nile8) FE a)dedt. 


On the other hand, note that K,y(x’) := K(x’,x)y € Y, and consider its Fourier transform 








(K(-.x)y)(E,2) =f fe tne Bete) (KW, a)y) (hdr 
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Using Equation (28) and the Plancherel formula, the integral on the right hand of the above equation 
is equal to 
e7 2mi(x,8) $(t) 


(1 +4716) (14.422 ||2||2)F 
However, the right-hand side of Equation (29) is identical to 


fa komme or t)(1-+4n?|6|?)(1 + 4a? ljal?) ded, 





(30) 





Putting (30) into the above equation, we immediately know that the reproducing property (29) holds 
true. This verifies that K is the reproducing kernel of the Hilbert space H. 

To prove the universality of this kernel, let Z be any prescribed compact subset of X, we define 
the Laplace kernel, for any x,t € R, by G(x,t) := el and the operator B : L?(IR¢) > L?(R¢) by 


Bg(r) := dee e!"Sle(s)ds, VreR?. 


Then, K(x,t) =e~ |B and moreover 


Ba — (1) 


drl. s 
(1+4n?||¢||7) 2 





By Theorem 12, it now suffices to prove that G is universal and B is positive definite. To this 
end, note that there exists cg such that 





e72" (x—1,6) 
G(x,t) | ramen 
Since the weight function ; FE mE is positive, G is universal according to Micchelli et al. (2003). 


To show the positive definiteness of B, we obtain from Equation (31) and the Plancherel formula 


that at jp 
Ê(T)| dT 
(Bg,8) p : 
Re (1+ 422) 


From this observation, the assertion follows. E 





We now discuss continuous parameterized multi-task kernels. For this purpose, let Q be a locally 
compact Hausdorff space and, for any © € Q, B(@) be an n x n positive semi-definite matrix. We 
are interested in the kernel of the following form 


K(x,t) )= | Go )(x,t)B(@)dp(@), Vx,t EX, (32) 


where p is a measure on Q. We investigate this kernel in the case that, for any œ € Q, G(@) is 
a scalar kernel with a feature representation given, for any x,t € X, by the formula G(@)(x,t) = 


(O(X), 0a(t)) y. Now, we introduce the Hilbert space W = L?(Q,W 29, p) with norm defined, 
for any f : Q — WEY, by 
112 = f Ovyo) 
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Next, we define a map ®: X > £(Y,L?(Q,W @Y,p)), for any x € X and w € Q, by 


P(x)y(®) := a(x) 8 (VV B()y). 


By an argument similar to that used just before Theorem 13, we conclude that K is a multi-task 


kernel and has the feature map ® with feature space W. 
We are ready to present a sufficient condition on the universality of K. 


Theorem 15 Let p be a measure on Q and for every œ in the support of p, let G(@) be a continuous 
universal kernel and B(@) a positive definite operator. Then, the multi-task kernel K defined by 
Equation (32) is universal. 


Proof Following Theorem 11, for a compact set Z C X suppose that there exists a vector measure 
u such that 


i, ba(x) @ VB(o)(du(x)) =0. 


Therefore, there exists a œ € support(p) satisfying fzo (x) ® \/B(@o) (du(x)) = 0. Equivalently, 
S290 (x) (\/B(@o)du(x),y) = 0 for any y € Y. Since we assume G(@o) is universal, appealing to 
the feature characterization in the scalar case (Micchelli et al., 2006) implies that the scalar measure 
(,/B(@o)du(x),y) = 0. Consequently, we obtain that u = 0 since y € Y is arbitrary. This completes 
the proof of this theorem. E 


Next we offer a concrete example of the above theorem. 


Example 5 Suppose the measure p over {0,°°) does not concentrate on zero and B(@) be a positive 
definite n x n matrix for each œ € (0,°°). Then the kernel K(x,t) = fg el B(@)dp(@) is a 
multi-task universal kernel. 


Further specializing this example, we choose the measure p to be the Lebesgue measure on 
[0,cc) and choose B(@) in the following manner. Let A be n x n symmetric matrices. For every 
œ > 0, we define the (i, j)-th entry of the matrix B(@) as e-¥, i, j € N,. Recall that a matrix A is 
conditionally negative semi-definite if, for any c; E€ R,i € Nn with Vien, ci = 0, then the quadratic 
form satisfies };; jen, CiAijc; < 0. A well-known theorem of I. J. Schoenberg (see, e.g., Micchelli, 
1986) state that B(@) is positive semi-definite for all œ > 0 if and only if A is conditionally negative 
semi-definite. Moreover, if the elements of the conditionally negative semi-definite matrix A satisfy, 
for any i,j € Ny, the inequalities A;; > 5(Aii +Aj;;) and Aj > 0, then B(@) is positive definite 
(Micchelli, 1986). With this choice of A, the universal kernel in Example 5 becomes 

K = : Vi,7 EN 
( eD) = Ta i,j © Ny. 


5.2 Transformation Kernels 


In this subsection we explore matrix-valued kernels produced by transforming scalar kernels. To 
introduce this type of kernels, let Y = R” , X be a Hausdorff space and T, be a map from X to 
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X (not necessary linear) for p € N,. Then, given a continuous scalar kernel G : XxX > R, we 
consider the matrix-valued kernel on X defined by 


K(x,t) := (GTD) y Yx t EX. (33) 


Proposition 16 Let G be a scalar kernel and K be defined by (33). Then, K is a matrix-valued 
kernel. 


Proof For any m €N, {yi : yi € R”,i € Nm} and {x; : x; € X,i € Nm} then 


Y: (yi, K (xix) yj) =} } ypivaiG( TpXi, Tigh) 
i,jeNm Pigs 
Since G is a scalar reproducing kernel on Z, the last term of the above equality is nonnegative, and 
hence K is positive semi-definite matrix-valued kernel. This completes the proof of the assertion. i 


We turn our attention to the characterization of the universality of K defined by Equation (33). 
To this end, we assume that the scalar kernel G has a feature map 0: X — W and define the 
mapping ®(x) : R” — W, for any y = (y1,.--,Yn) E R”, by B(x)y = Lpen, Yp(Tpx). Its adjoint 
operator B(x)* : W — R” is given, for any w E€ W, as ®*(x)w = ((0(T1x),w) q,---, (O(Tnx),W) qy)- 
Then, for any x,t € X, the kernel K(x,t) = &*(x)@(t) and thus, we conclude that W is the feature 
space of K and ® is its feature map. 

We also need some further notation and definitions. For a map T : X —> X, we denote its range 
space by TX := {Tx : x € X} and T~!(E) := {x : Tx € E} for any E C X. In addition, we say that 
T is continuous if T7! (U) is open whenever U is a open set in ra Finally, for any scalar Borel 
measure v on X and a continuous map T from X to X, we introduce the image measure vo T7! on 
X defined, for any E € A(X), by (vo T7!) (E) :=v({x EX: Tx € E}). 

We are ready to state the result about universality of the kernel K in Equation (33). 


Proposition 17 Let G be a scalar universal kernel, T, : X — X be continuous for each p € Nn and 
define the kernel K by Equation (33). Then K is universal if and only the sets TX, q € Nn, are 
pairwise disjoint and T} is one-to-one for each q € Ny. 


Proof Following Theorem 11, for any compact set Z C X, it suffices to verify the equation 
@(Z)+ = {0}. Before doing so, we recall that, by Lemma 10 and the remark which followed 
it, for any vector measure u E€ M (Z, IR"), there exists a scalar regular measure v € M (Z x Bı) such 
that 


du(t) = (favor f, ave»). 


Hence, any vector measure u can be represented as u = (u1, . - . , Un) where each u; is a scalar measure. 
Then, u € &(Z)+ can be rewritten as 


E [Todu = 


geENn 


Equivalently, if Z:= Ugen, Tg Z we conclude that 


[oa > mo T) =0; 


qENn 
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Since T} is continuous for any q € N,, the range space 7, Z is compact and so is Z. Recall from 


Micchelli et al. (2006) that the scalar kernel G is universal on Z if and only if its feature map 9 is 
universal on Z . Therefore, the above equation is reduced to the form 


-1 
Ł MgoT, =0. 
qENn 


Consequently, we conclude that K is universal if and only if 


{(u1,-.-,4n) : Ł ugo T; =O} ={0}. (34) 


qENn 


With the above derivation, we can now prove the necessity. Suppose that {T,X : q € Nn} is not 
pairwise disjoint. Without loss of generality, we assume that 71X O 7>X # 0. That means there exists 
X1,X2 € X such that Tixı = Pox = zo. Let uq = 0 for q > 3, and denote by 6,—y the point distribution 
at x’ € X. Then, choosing yw, = d,=x,, and u2 = —8,—,, implies that Equation (34) holds true. By 
Theorem 11 in Section 4, we know that K is not universal. This completes the first assertion. 

Now suppose that there is a map, for example Tp, which is not one-to-one. This implies that 
there exists x1,x2 E X, x; Æ x2, such that Tpxı = Tpx2. Hence, if we let ug = 0 for any q 4 p and 
Mp = Sx=x, — Sy=x, then Yoen, Ug O To = 0,. But up 4 0, hence, by Theorem 11, K is not universal. 
This completes the our assertion. 

Finally, we prove the sufficiency. Since py © T; ' only lives on T} X and {T,X : q € Ny} is 
pairwise disjoint, then Yoen, Mg © T = 0 is equivalent to pg © T; ' = 0 for each q € N,. How- 
ever, since T} is one-to-one, E = T;'(T,(E)) for each Borel set E € A(X). This means that 
AE) = 44° T (T}(E)) = 0 for any E € A(X). This concludes the proof of the proposition. W 


We end this subsection with detailed proofs of our claims about the examples presented in 
Section 2. Indeed, we already proved the positive semi-definiteness of the kernel in Example 2 by 
Proposition 16. Below, we prove the claim that the function K given by Equation (6) is not a kernel 
in general. 


Proposition 18 Let Op, > 0 and Opg = Oqp for any p,q € Nn. Then, the matrix-valued function 
defined by 


n 
K(x,t) = (eral) , Vx,t EX 
p.q=l 
is a multi-task kernel if and only if for some constant ©, © pg = © for any p,q © Nn. 


Proof When (Opa). 4 is a constant matrix then K is positive semi-definite. Conversely, suppose 
K is a multi-task kernel which means, for any m € N and x; € X with i € Nn, that the double-indexed 
nm X nm matrix 


(Cp), Ga) = embers?) (35) 


ip), 4) Nm xNn 


is positive semi-definite. 
We choose any distinct positive integers po and go. In Equation (35), we specify any m,n with 
m > n such that po,go E Nm, X1,---,%n With Xpo A Xq and set c = ||xp, Sxl: Therefore, we 
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conclude that the matrix 


1 1 EXP{—COpo py} EXP{ —CO pogo } 

1 1 EXP{=COg pp} EXP{=COgoan 
EXP{—COpopy)} EXP{—CO pogo } 1 1 
EXP{—COg py} EXP{ —CO gogo } 1 1 


is positive semi-definite. Consequently, the determinant of its 3 x 3 sub-matrix in the upper right 
, wi ; 

hand corner, which equals — [exp{—cO popo } — €XP{—cOgopo}] ,» is nonnegative. Therefore, we con- 

clude that © popo = Oqopo- E 


5.3 Hessian of Gaussian Kernels 


In this subsection we consider the universal example of the Hessian of scalar Gaussian kernels 
(Example 3 in Section 2). To introduce this type of kernels, we let X = Y = R” and G be a 
Gaussian kernel with deviation o, that is, for any x € R”, G(x) = e-lxl?/o with o > 0. Then, the 
Hessian matrix of the kernel G is 
n 
K(x,1) = ( — @p0qG)(x—1)) Yate". (36) 
P.q= 


and so alternatively, K has the form 
K(x,t) = 4n(2n0)"” ii eiat E) get eoll? ge (37) 
R” 


Corollary 19 Letn > 1 and K : R” x R" — R"*" be defined by (36). Then, K is a matrix-valued 
kernel which is universal if and only ifn = 1. 


Proof The fact that K is positive semi-definite directly follows from the observation, for any m € N, 
{yi : yi € R”,i € Nm} and {x; : x; € X,i € Nm}, that 


2 


F nk yerm6)] oll ae, 


i€Nmn 


E (iK(i,x))yj) = 4n(2n0)"" i 


i, j,E€ENm 








In order to prove the universality of K, we follow Theorem 11. For this purpose, we assume that 
Z is a compact subset of X and u € K(Z)", that is, 


f K(x,t)(du(t)) WE (38) 
Z 
By Equation (37), this equation is equivalent to 
J emir) teoll | emit) (E du(t)\dé = 0, Vx € Z, 
" Z 


which implies, by integrating both sides of this equation with respect to x € R”, that 


f eok 


2 eS), du(t)) “at =0. 
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Consequently, Equation (38) is identical to the equation 


iB emit) (E du(t))=0, VEER". 


If n = 1, the above equation means that 
| et du(t) = 0, VEER. 
Z 


Taking the k-th derivative with respect to € of both sides of this equation and set & = 0, we have, for 
every k € N, that 


1 t*du(t) =0. 
Z 


Since polynomials are dense in C(Z), we conclude from the above equation that u = 0. Hence, by 
Theorem 11, the kernel K is universal when n = 1. 

If n > 2, we choose ug = 0 for q > 3 and du (t) = dti (8,=1 — 8 WIT, 35;,=0 and duz (t) = 
(8,,=—1 — õn =1 dh Ip- 5;,=0, and note that 








sin(2m&}) 


~2ni(t,§) q t) = (—2misin(27 
age Be getaan) Ti 


and 
l in(2 
1 e208) dm(t) = (2misin(2n)) ee. 
[-11p n2 


Therefore, we conclude that 


Tan E (Ecdat) = SI | 


api mO pe Pde = 0. 


[-1,1]” 


Hence, the kernel K is not universal when n > 2. E 


5.4 Projection Kernels 


In the final subsection we introduce a class of multi-task kernels associated with projection operators 
of scalar kernels. 

We start with some notation and definitions. Let X C Rf, Q C R” be a compact set and 
Y =L (Q). We also need a continuous scalar kernel G : (X x Q) x (X x Q) :— R with a fea- 
ture representation given, for any x,x' € X and t,s € Q, by 


G((x,t), (x',s)) = (0(x,t),0(x",5)) w. (39) 


Then, the projection kernel K : X x X — L(Y, Y) is defined, for any f € L*(Q), by 
(Kx) f(t) = f G((x,t), (x',s))f(s)ds, Vx,x EX, tER. (40) 
Q 
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We first show that K is a multi-task kernel. To see this, for any m € N, {x; : x; € X,i € Nm}, and 
{yi : yi € L?(Q),i € Nm} there holds 


ijen, (Kanay) =Lijerin fof Gl ait), (47-8))v4(s)yilt)deds 
2 
f ansias 


= LicNn 20 


zj 














which implies that K is a kernel. 


To investigate its universality from the feature perspective, we define the mapping ® : X — 
L(Y, W), for any x € X and y E€ Y, by 


By = [Ox s)y(o)ab 


and also its adjoint operator ®* is given, for any w E€ W, by ®*(x)w = (0(x,-),w) w. Hence, for any 
x,x’ € X, we conclude that K (x,x’) = &*(x)®(2x’) which implies that K is a multi-task kernel and ® 
is its associated feature map. 

Our next goal is to prove the universality of K. 


Theorem 20 Let G and K be defined as in Equations (39) and (40). If G is a universal scalar kernel 
then K is a universal multi-task kernel. 


Proof By Theorem 11, it suffices to show that, for any compact Z C X, whenever there exists a 
vector measure u such that 


[209 (dutx) =0, 


then u = 0. Note that u is an L?(Q)-valued measure. Hence, u can alternatively be interpreted as a 
measure p(-,-) on Zx Q defined, for any E € @(Z) and E' € A(Q), by u(E,E') := | u(E)(s)ds. 
E' 


From this observation, we know that 


[E= f f odus) 


Since Z and Q are both compact, then Z x Q is also compact by Tychonoff theorem (Folland, 1999, 
p.136). By assumption, G is universal on X x Q and ọ is its feature map, and thus we conclude that 
the scalar measure du(x,s) is the zero measure. This means that, for any E € Z(Z) and E' € BQ), 


f u(E)(s)ds = 0. 
a 


Since E,E’ are arbitrary, we conclude that the vector measure u = 0 which completes the assertion. 
a 
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6. Conclusion 


We have presented a characterization of multi-task kernels from a Functional Analysis perspective. 
Our main result, Theorem 4 established the equivalence between two spaces associated with the 
kernel. The first space, Cx(Z, 9), is the closure of the linear span of kernel sections; the second 
space, Co(Z,%), is the closure of the linear span of the features associated with the kernel. In 
both cases, the closure was relative to the Banach space of continuous vector-valued functions. This 
result is important in that it allows one to verify the universality of a kernel directly by considering 
its features. 


We have presented two alternate proofs of Theorem 4. The first proof builds upon the work 
of Micchelli et al. (2006) and the observation that a multi-task kernel can be reduced to a standard 
scalar-valued kernel on the cross product space Z x Y. The second proof relies upon the theory of 
vector-measures. This proof is constructive and provides necessary and sufficient conditions on the 
universality of a multi-task kernel. They are summarized in Theorem 11, which is our main tool for 
verifying the universality of a multi-task kernel. 


In both proofs, an important ingredient is a principle from Functional Analysis, which uses the 
notion of the annihilator set. This principle, which is a consequence of the Hahn-Banach Theorem, 
states that two closed linear subspaces of a Banach space—in our case Cx(Z,9) and Co(Z,”)— 
are equal if and only if whenever a bounded linear functional vanishes on one of them, it also 
vanishes on the other one. 


A substantial part of the paper has been devoted to present several examples of multi-task ker- 
nels, some of which are valuable for applications. Although much remains to be done on developing 
applications of the theory of universal kernels, we hope that our theoretical findings, as they are il- 
lustrated through the examples, will motivate further work on multi-task learning in applied machine 
learning. 
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Appendix A. 


This appendix gives the proof of Lemmas 9 and 10 in Section 4. 
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Proof of Lemma 9 By the definition of the integral appearing in the right-hand side of equation it 
follows (17) (see, e.g., Diestel and Uhl, Jr., 1977), for any f € C(Z,%), that 


[Lu] < lall sup || F(x) lly- 
xEZ 


Therefore, we obtain that ||L,,|| < ||u||, and thus L, € C* (3,9). 

To show that ||L|| = ||yu||, it remains to establish that ||u|| < ||L,|]. To this end, for any £ > 0 
and, by the definition of |||], we conclude that there exist pairwise disjoint sets {A}; : j € Nn} 
such that Ujen,Aj C Z and |u|] = |u|(Z) < € + Lien, ||u(Aj) ll. We introduce the function g = 


Lien, kae D Xa which satisfies, for any x € Z, the bound ||g(x)||y < 1. Since |u| is a regular 
measure on Z, applying Lusin’s theorem (Folland, 1999, p.217) to the function %4,, there exists a 
real-valued continuous function f; € C(Z) such that |fj(x)| < 1 for any x € Z and f; = %4, except 
on a set E; with |u|(Ej) < Gaya We now define a function h : Y — Y by setting h(y) = y, 


if ||yl|y < 1 and tas j= Diy if ||yl|y > 1, and introduce another function in C(Z,%) given by 


f:= = Lien, ale ti f;. Therefore, the function f = hofisi in C(Z, Y) as well, because f E C(Z,Y) 
and, for any y, y € Y, |A) —AO’)lly < 2lly—y'||y- Moreover, we observe, for any x € (U jen, Ej i“ i 
that f(x) = g(x) and, for any x € Z, that || f(x)||y < 1. 

We are now ready to estimate the total variation of u. First, observe that 


[ lf) sly dal) < E + DME) <s 


JENn 


and consequently we obtain the inequality 


lall < Ejen, lat r i (s6),dulx)) + 


<| S Fe) -80), |+| fre +e 


< ff) — 8) ldlal(x ee 


This finishes the proof of the lemma. a 
We proceed to the proof of Lemma 10. 


Proof of Lemma 10 For each u E€ M(X,), there exists an L, E C*(Z,%) given by Equation (17). 
The isometry of the map u —> Ly follows from Lemma 9. 

Therefore, it suffices to prove, for every L € C*(Z,9), that there is a u € M(Z,%) such that 
L,, = L. To this end, note that Lo1~' € C*(Z x Bı) since 1 is an isometric map from C(Z,%) 
onto G(Z x Bı) defined by Equation (10). Since C,(Z x Bı) is a closed subspace of C(Z x B1), 
applying the Hahn-Banach extension theorem (see, e.g., Folland, 1999, p.222) yields that, for any 
L € C*(Zx Bı), there exists an extension functional L € C*(Z x Bı) such that (F) = Lov! (F) 
for any F € G(Z~x Bı). Moreover, recalling that Z x Bı is compact if Bı is equipped with the weak 
topology, by the Riesz representation theorem, for any L, there exists a scalar measure v on Z x By 
such that 


HS [og EEDE), YF € C(Zx By). 
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Equivalently, for any f € C(Z,%) there holds 


Lf =Lov'(F)= f 


ZX Bı 


F(xy)dv(x,9) = | Cda) = Lyf 


where u is defined i terms of v as in Equation (15). 
This finishes the identification between functionals in C(Z, 9’) and vector measures with bounded 
variation. a 
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